Repository navigation
Ease of use - #4
Merged
Merged
Conversation
Add a retraining-free expert-pruning path for the MoE backbone, complementing quantization as a size lever. - zonos2-perplexity --prune-stats: export per-(layer,expert) hit count + MSAN (mean ‖down-input‖² over routed tokens, the (1,0,2) activation-norm score from arXiv:2606.15716), reusing the existing imatrix accumulators. - prune-cli: gguf→gguf rewrite that gathers the kept experts (and matching router_mlp4/router_bias slices) into a smaller GGUF; pure per-expert memcpy so it composes with quantize-cli in either order. - Runtime --prune-mask on zonos2-cli/zonos2-perplexity: apply a keep-set dynamically (additive -inf mask before softmax AND on the top-k scores), for sweeping the quality knee without rewriting GGUFs. Byte-identical to a baked prune-cli model. - prune-policy.h: shared keep-set selector with four modes — --keep N (uniform), --mass-eps E (relative MSAN-mass threshold, non-uniform), --drop-lowest N (global lowest-MSAN), --drop-below-hits H; floored at each layer's top_k. - build_moe reads the expert count per layer from ffn_up_exps->ne[2] (not hp.n_expert), so heterogeneous pruned models load and the softmax renormalizes over survivors. Finding on this model: pruning degrades gracefully (~0.02 KLD/expert, no cliff) but is dominated by quantization on quality-per-byte; non-uniform mass-targeting is worse than uniform because top_k=1 makes per-layer routing diversity the binding constraint. Tooling kept for future, less MoE-sensitive models. Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>
Add a runtime whole-block bypass for measuring depth (layer) pruning,
complementing the expert (width) pruning path. `zonos2_model.layer_skip`
marks transformer blocks to skip at graph-build time: build_moe/attention
are not built and the residual stream passes through unchanged (exact
identity), so both compute and the block's KV-cache are saved. Exposed via
`zonos2-perplexity --skip-layers L1,L2,...` for leave-one-out / span KLD
sweeps against a --kl-divergence base.
Phase-0 leave-one-out on out/zonos2-f16.gguf (28 blocks, vs ref-multi):
the profile is literature-consistent in shape (shallow blocks 0-2 and the
final block 27 catastrophic to drop, KLD 2.4-5.6; deep MoE cluster 21/23/
25/26 most redundant, KLD 0.23-0.28) but dominated by quant in magnitude —
the single most-droppable block (L25) costs KLD 0.229 (~q3k-cliff level)
for -3.6% blocks, vs q4k's 0.072 for -72% expert bytes. Generative KLD
collapses as the depth-pruning literature's MC-benchmark wins predict it
would ("Benchmark Illusion"), and there is no healing stage here. Knob
kept for future less-MoE-sensitive models; pruning stays parked.
Co-Authored-By: Claude Opus 4.8 <[email protected]>
…GUF discovery) Release archives now bundle a start-zonos2 launcher (.sh for Linux, .command for macOS Finder, .bat/.ps1 for Windows) that downloads any missing GGUFs from the now-public Zyphra/ZONOS2-GGUF repo via system curl (resumable, q6_k default), launches zonos2-server with the right flags, and opens the web UI; CI sed-stamps the backend default to gpu in the vulkan/metal archives and the Windows artifact becomes a .zip for Explorer-native extraction. Both binaries auto-detect dac.gguf / spk-encoder.gguf next to the backbone (gated in the CLI so .npy outputs keep their behavior), accept --spk and --spk-encoder as aliases, and print an actionable curl hint when a model file is missing. New tests/launcher-smoke.sh covers download-skip, download from a local http server, the optional-spk warning, and teardown on linux/macos CI; the windows job parse-checks the ps1. Co-Authored-By: Claude Fable 5 <[email protected]>
…bview) Opt-in -DZONOS2_APP=ON target that runs the whole HTTP server in-process on an ephemeral 127.0.0.1 port and points a native webview (WebView2 / WKWebView / WebKitGTK, via saucer v8) at it, so the app UI is the exact web/tts_ui.html the server already serves and the HTTP API stays reachable while the app runs. First launch shows a setup page with native .gguf pickers; the choice is saved to the platform config dir and later launches boot straight in. Server-style CLI args pass through verbatim for power users. server.cpp: main() split into server_run(...,ctl) + zonos2_server_main(); real main() guarded by ZONOS2_APP_EMBED. --port 0 now binds an ephemeral port (bind_to_any_port/listen_after_bind) and publishes it back through a zonos2_server_ctl so the app can attach and stop the server. Transparent to normal server/CLI use (readme-commands + launcher-smoke still green). CI: new linux-app job (ubuntu-24.04, gcc-14, GTK4>=4.12); macos bumped to macos-15 + Xcode>=16.3 and builds/signs Zonos2.app; windows bundles zonos2-app.exe; release waits on linux-app. saucer's runtime SQLite state (.saucer) is gitignored. Toolchain floor is much higher than the rest of the project (CMake>=3.31, C++23); everything else keeps the C++17/CMake 3.18 floor. Validated by building zonos2-app in an ubuntu:24.04 container and driving it headlessly (xvfb+dbus): ephemeral bind, /health, and /tts/generate through the embedded server all pass. Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>
saucer's WKWebView backend and saucer-embed's injected sources are Objective-C++. saucer enables OBJC/OBJCXX inside its own FetchContent subdirectory, but that scope doesn't set CMAKE_OBJCXX_COMPILE_OBJECT for the top-level directory where zonos2-app is defined (embed injects .mm sources into it), so CMake fails at the generate step with "CMAKE_OBJCXX_COMPILE_OBJECT not set" — macOS only (WebKitGTK is plain C++, so Linux/Windows were unaffected). Enable OBJC/OBJCXX in the top-level scope under -DZONOS2_APP=ON, guarded to APPLE. Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>
The previous fix (683fb2d) enabled OBJC/OBJCXX inside the ZONOS2_APP block, but that runs *after* add_subdirectory(ggml). ggml's Metal backend enables OBJC in its own subdirectory scope first, so the later top-level enable_language(OBJC) no-ops and CMAKE_OBJC_COMPILE_OBJECT is never set in the top-level scope where zonos2-app is defined -> generate still failed, now on OBJC (OBJCXX was fine because ggml never enables it, so our call was its first enabler). Move the option() + the APPLE enable_language(OBJC/OBJCXX) above add_subdirectory(ggml) so the root scope is the first enabler of both languages; the compile rules then propagate down to ggml and to zonos2-app alike. Still APPLE- and ZONOS2_APP-guarded, so Linux/Windows and non-app builds are untouched (native reconfigure clean). Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>
Voice cloning decodes the reference audio through ffmpeg; plain TTS never needs
it. Instead of requiring users to install ffmpeg, resolve it as $ZONOS2_FFMPEG
-> a binary next to the executable -> PATH (spk-encoder.cpp, via a new shared
exe-path.h), and have the start-zonos2 launcher and the desktop app download a
static ffmpeg on first run when none is present.
Sources (fetched at the user's runtime from the provider — nothing is
redistributed, and ffmpeg runs as a separate process): Linux/Windows use the
BtbN LGPL builds (ffmpeg-n7.1-latest-{linux64,win64}-lgpl, extracted from the
archive); macOS arm64 has no clean LGPL static build, so a pinned GPL binary
(eugeneware/ffmpeg-static b6.1.1 ffmpeg-darwin-arm64). Override with
ZONOS2_FFMPEG (a path) or ZONOS2_FFMPEG_URL (a single-binary source).
The launcher fetches into <model-dir>/bin and exports ZONOS2_FFMPEG; the app
fetches into its config dir on a detached thread so startup isn't blocked
(cloning fails gracefully until it lands). A system ffmpeg on PATH is used as-is
(ZONOS2_SKIP_PATH_FFMPEG=1 forces a download, for tests). Standalone zonos2-cli
does not auto-download — it resolves env/next-to-exe/PATH.
Validated: native resolution (env/PATH/bogus) + a real clone decode; launcher
ffmpeg-download smoke (single-binary + tar-extract paths); and an ubuntu:24.04
container build of zonos2-app with a headless auto-download into the config dir.
Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>
exe-path.h pulls <windows.h>, and app.cpp includes it before <httplib.h>'s <winsock2.h>. Left alone, <windows.h> drags in the legacy <winsock.h> (v1), which hard-conflicts with <winsock2.h> — hundreds of sockaddr/fd_set redefinition errors, Windows-only (server.cpp is immune because compat.h never pulls windows.h, so httplib includes winsock first there). Define _WINSOCKAPI_ (+ NOMINMAX, matching httplib) before <windows.h> in exe-path.h. Use _WINSOCKAPI_ rather than WIN32_LEAN_AND_MEAN so the OLE/COM headers saucer's WebView2 backend needs stay available. Also drop app.cpp's now -dead <windows.h>/<mach-o/dyld.h> block (exe-path.h provides them). Linux app build + ffmpeg smoke reconfirmed in the ubuntu:24.04 container. Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Improve packaging and usage for laymen users